Skip to content

[DO NOT LAND] Allow us to disable namespace listwatching #1

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 305 commits into from

Conversation

imbstack
Copy link

@imbstack imbstack commented May 8, 2025

The namespaces cache will be an empty store in this case. Down the line all of the code should just work (it appears to check for missing namepaces, etc) for instance emitting an error log if you try to access namespace labels.

Don't land this yet. I still want to add tests, etc. Maybe we can try this on staging though. @mdbenjam were you able to use an image you built on your local machine when you were patching vector before?

Relates to renderinc/api#23296

@imbstack imbstack requested review from mdbenjam and hieurender May 8, 2025 21:53
@hieurender
Copy link

excited to see what this will do in staging

@mdbenjam
Copy link
Collaborator

mdbenjam commented May 9, 2025

@imbstack I think I never got that far. I ultimately got pulled in another direction and never finished my patch.

thomasqueirozb and others added 26 commits June 12, 2025 14:20
…22899)

* Set ratelimitng as default

* Remove explicit internal_log_rate_limit flag

* Add tests

* Review comments

* add internal ratelimit only if the field is  present

q

---------

Co-authored-by: Pavlos Rontidis <[email protected]>
…ordotdev#23196)

* feature(kafka sink): add rate limit configuration for `kafka` sink

Related: vectordotdev#21879

* Add changelog entry

---------

Co-authored-by: Pavlos Rontidis <[email protected]>
…22964)

* allow to provide aws session token

* doc generation

* Update changelog.d/aws-auth-session-token.enhancement.md

Co-authored-by: Pavlos Rontidis <[email protected]>

* incorporate comments

* rerun make generate-component-docs

* fix bare url in doc

---------

Co-authored-by: Pavlos Rontidis <[email protected]>
… line size to be applied after merging instead of just before (vectordotdev#22582)

* Add config for maximum allowed line size after merging

* Add warns when we drop partial logs for being too big; shift some comments around

* Add changelog

* Format

* Increment component_discarded_events_total on violation of max_line_size and max_merged_line_size

* Update changelog.d/22581_max_merged_line_bytes.feature.md

Co-authored-by: Pavlos Rontidis <[email protected]>

* Don't emit expired events that are too big nor ones that don't appear to be partial; fix test

* Fix another test

* Update src/sources/kubernetes_logs/mod.rs

Co-authored-by: Pavlos Rontidis <[email protected]>

* Update src/sources/kubernetes_logs/mod.rs

Co-authored-by: Pavlos Rontidis <[email protected]>

* Remove inadvertently added file

* Include Value rather than Event in error struct

* Rename field in bucket struct

* Move max_merged_line_bytes from being a param to being a field on the state struct

* Make new config field optional, defaulting to old behavior

* Format

* Appease check-events

* docs regen

* Tweak wording of doc; emit only first 1k bytes of dropped lines in error

* Rename fields for clarity

* Per PR feedback: copy just the initial 1000 bytes rather than cloning the whole buffer and then truncating, use more idiomatic Rust for handling both configured and unconfigured cases of max_merged_line_bytes

* Allow spelling of already-merged changelog filename

* Don't try to include more characters than there actually are in the slice

* Don't just get enough capacity, make sure length matches too

* Formatting

---------

Co-authored-by: Orri Ganel <[email protected]>
Co-authored-by: Pavlos Rontidis <[email protected]>
…rdotdev#22971)

* fix(amqp sink): attempt one reconnect when channel has errored

* add changelog.d

* fix changelog.d

* Use vector_common::Error

* use deadpool to manage channel recovery and multiple channels

* Rename AmqpSink

* Update integration tests

* make amqp max_channels configurable; default to 4; added to docs and changelog

* dupe word

* Update 3rd party licenses for deadpool

* Fix formatting in amqp.cue

* Update DEVELOPING.md
Bumps [rustls](https://github.com/rustls/rustls) from 0.20.9 to 0.22.4.
- [Release notes](https://github.com/rustls/rustls/releases)
- [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md)
- [Commits](rustls/rustls@v/0.20.9...v/0.22.4)

---
updated-dependencies:
- dependency-name: rustls
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
… panic (vectordotdev#23201)

* fix(opentelemetry source): Handle NaN without panic

* Update changelog.d/23199_opentelemetry_source_panic_on_nan_attributes.fix.md

* Update 23199_opentelemetry_source_panic_on_nan_attributes.fix.md

Remove trailing space

---------

Co-authored-by: Pavlos Rontidis <[email protected]>
…rce.md) (vectordotdev#23166)

Fix config extension typo in Windows command (from-source.md)

Corrects the configuration file extension in the Windows startup command for Vector. 
The original command `.\target\release\vector --config config\vector.toml` incorrectly used TOML extension, 
while Vector's actual config file is YAML format.
…rdotdev#21293)

* fix(elasticsearch): Encode bulk action parameters as JSON

They are currently using string templating which, if there are special characters in the value, will
end up creating an invalid JSON payload; an issue that can be difficult to track down. This happened
in vectordotdev#21288.

Signed-off-by: Jesse Szwedko <[email protected]>

* Fix tests

Signed-off-by: Jesse Szwedko <[email protected]>

* Use serde_json to check json equality

* Remove unnecessary &

* Unwrap instead of ignoring error

---------

Signed-off-by: Jesse Szwedko <[email protected]>
Co-authored-by: Thomas Schneider <[email protected]>
…ctordotdev#23151)

* fix(aws_ecs_metrics): skip over empty ECS metrics payloads

* chore: add changelog entry

* Update changelog.d/skip_empty_ecs_metrics.fix.md

---------

Co-authored-by: Pavlos Rontidis <[email protected]>
…tordotdev#23213)

* fix(ci): add missing perms to gardener_remove_waiting_author.yml

* add fail on error

* only run if label exists

* Update .github/workflows/gardener_remove_waiting_author.yml

* Update .github/workflows/gardener_remove_waiting_author.yml

* Update .github/workflows/gardener_remove_waiting_author.yml

* Update .github/workflows/gardener_remove_waiting_author.yml

---------

Co-authored-by: Thomas <[email protected]>
…tordotdev#23204)

* feat(website): render input and output badges for all components

* fix badges

* v1 display traces in output data

* remove printfs

* fix badge rendering
…tdev#23214)

* Remove random log file

* Remove enableable from expect.txt since it is in allow.txt
…dotdev#23216)

* docs(external): fix memory enrichment table highlight wording

* improve wording
chore(docs): fix opentelemetry typo

Co-authored-by: Pavlos Rontidis <[email protected]>
* chore(website): prettier rc formatting of hugo/html

* increase width to 120
* Make interval_ms {read,write,delete}able

* Add interval_ms to VALID_METRIC_PATHS_SET

* Add backtics to docs, clippy complains

* Add changelog

* Rename changelog to add .feature

* Update changelog.d/23217_interval_ms_modifiable.feature.md

Co-authored-by: Pavlos Rontidis <[email protected]>

* Move struct Case up

* Fix clippy again

* Fix test

---------

Co-authored-by: Pavlos Rontidis <[email protected]>
…ectordotdev#23207)

* chore(config): Fix `InlineSingleUseReferencesVisitor` failing tests

The above visitor is intended to inline schemas in the references section for
which there is only a single reference. This visitor is, however, unable to
detect some cases where there are in fact multiple references, ending up
inlining the schema multiple times. This has led to bloat in generated schemas.

In the core Vector schema, this affects 121 structures, shaving 844 lines off of
the generated schema.

* Regenerate component docs
…rdotdev#22706)

* enhancement(http_client source): allow VRL in query parameters

* update semantic scopes

* chore: update cue docs for http_client

* refactor: handle VRL compilation and regex during build

* chore: fix punctuation for message

* refactor: use vrl type in query parameter instead of regex

* chore: add better VRL result -> string handling

* chore: update docs

* chore: use raw identifier for type

* chore: allow missing const for value

---------

Co-authored-by: Pavlos Rontidis <[email protected]>
Co-authored-by: Thomas <[email protected]>
…inks (vectordotdev#23222)

* external(docs): mention that encoding config can change what telemetry types are supported

* feat(website): add input telemetry types section

* remove weird char

* visual improvements
…ev#23221)

* Update to_alternative_components function signature from VRL

* Simplify + add interval_ms to mut code path

* Fix clippy warnings

* cargo vdev build licenses
thomasqueirozb and others added 23 commits August 7, 2025 14:37
This standardizes the last step in many workflows that were only checking for failure and missed the
check for cancelled
* chore(codecs protobuf): allow all telemetry types

* changelog

* fix
* chore(deps): cargo update -p vrl

* chore(dev): cargo vdev build licenses
…dev#23543)

* Reapply "Remove downcast ref and trait usage from partial retry logic"

This reverts commit 3bd9c0d.

* Reapply "Fix make check-clippy errors"

This reverts commit abeb520.

* Use Fn instead of FnOnce
…fault (vectordotdev#23566)

* Revert "feat(dev): Enable `internal_log_rate_limit` by default (vectordotdev#22899)"

This reverts commit dffcb5a.

* added internal_log_rate_limit = true to new traces

* cargo fmt

* Fix compilation errors and remaning missed logs

* Checkout file from dffcb5a~1

* Fix clippy

---------

Co-authored-by: Thomas <[email protected]>
* chore(ci): skip int tests on website only PRs

* Fix script

* Add dedicated error message
* chore(releasing): prepare v0.49.0 release (vectordotdev#23563)

* chore(releasing): Pinned VRL version to 0.26.0

* chore(releasing): Generated release CUE file

* chore(releasing): Updated website/cue/reference/administration/interfaces/kubectl.cue vector version to 0.49.0

* chore(releasing): Updated distribution/install.sh vector version to 0.49.0

* chore(releasing): Add 0.49.0 to versions.cue

* chore(releasing): Created release md file

* Apply suggestions from code review

* edit

* added description

* authors

* add upgrade guide wip

* upgrade guide changes

* Apply suggestions from code review

Co-authored-by: Pavlos Rontidis <[email protected]>

* improve breaking guide

---------

Co-authored-by: Thomas <[email protected]>

* cargo vdev build manifests

* Bump version to next release

---------

Co-authored-by: Pavlos Rontidis <[email protected]>
…otdev#23577)

* unrelated web-playground build fix

* ran cargo fmt
)

* chore(website): 0.49.0 release - address docs review

* Update website/content/en/highlights/2025-08-12-0-49-0-upgrade-guide.md

Co-authored-by: Thomas <[email protected]>

---------

Co-authored-by: Thomas <[email protected]>
…ctordotdev#23580)

* chore(opentelemetry source): config module and reduce duplication

* ran cargo fmt

* fix and reorganize imports

* ran cargo fmt
* chore(deps): bump async-nats from 0.33.0 to 0.42.0

* chore: update license
* chore(ci): sort semantic scopes

* trailing spaces
The namespaces cache will be an empty store in this case. Down
the line all of the code _should_ just work for instance emitting
an error log if you try to access namespace labels.
@imbstack imbstack force-pushed the bstack/vector-22990 branch from dc230ad to 1d09322 Compare August 15, 2025 04:09
@@ -0,0 +1,4 @@
Allow disabling listwatching of Kubernetes namespaces to reduce resource usage in clusters with

Check failure

Code scanning / check-spelling

Unrecognized Spelling Error

listwatching is not a recognized word. (unrecognized-spelling)
@imbstack imbstack closed this Aug 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.